DOCUMENT RESUME 

ED 360 925 HE 026 663 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Cash, R. William 

Assessment of Study- Abroad Programs Using Surveys of 
Student Participants, AIR 1993 Annual Forum Paper. 
May 93 

24p.; Paper presented at the Annual Forum of the 
Association for Institutional Research (33rd, 
Chicago, IL, May 16-19, 1993)- 

Information Analyses (070) ~ Reports - Descriptive 
(141) — Speeches/Conference Papers (150) 

MFOl/PCOl Plus Postage. 

College Outcomes Assessment; College Programs; 
College Students; '^Data Collection; Evaluation 
Methods; Higher Education; Institutional Research; 
Outcomes of Education; ^^Program Effectiveness; 
^Student Reaction; ^Study Abroad; ^^Surveys; 
Undergraduate Study 
^AIR Forum; '^Saint Marys College IN 



ABSTRACT 

Study-abroad programs are among the many academic 
programs being evaluated in the assessment of colleges and 
universities. One evaluative tool is the survey of students who 
participate in these programs. This paper di'-^cusses the use of such 
surveys by St. Mary's College (Indiana) durin; the past decade to 
gather information regarding program strengths and weaknesses and 
assess the impact of the study abroad program on participants. 
Statistical data are presented showing what St. Mary*s College has 
learned about its study-abroad program and student needs over the 
yearr. . These data were collected not only from student surveys of 
those completing their study-abroad programs, but also routinely from 
incoming freshmen as well as from outcome measures administered 
during and after the college experience. While the surveys provide 
compelling evidence of the immediate effects of these programs, 
long-term effects were found to be more difficult to determine. Part 
of the value in assessing these programs has been enhancing the 
school's ability to make program changes and modifying the program 
experience and student preparation to achieve greater positive 
impacts. (Contains nine references.) (GLR) 



Reproductions supplied by EDRS are the best that can be made 
>v from the original document. 



Assessment of study-abroad programs 

1 



ASSESSMENT OF STUDY-ABROAD PROGRAMS USING 
SURVEYS OF STUDENT PARTICIPANTS 



R. William Cash 
Director 
Office of Institutional Research 
Saint Mary's College 
Notre Dame, Indiana 46556 
(219) 284-4574 



U S DCFArrnCMT Of EIXJCATIOM 

Ontcc ot Educ^liOnAI RewarCh andlmp(ovem«nt 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 



rThis (Jocumeni has t>een reprodoced as 
received from ihe person Of organizalion 
Originating i| 
n Minor changes have been m«<fe 10 improve 

reproduction Quality 

• Points of view 0* opinions Staled >n this docu 
ment do not necessarity represent offiCtai 
OERl positfon or policy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



AIR 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)," 



Presented at 

Association for Institutional Research Annual Forum 
Chicago, Illinois 
May 18, 1993 



^ RESTCOPVAVAIIABLE 



for Management Research, Policy Analysis, and Planning 



•x'his paper was presented at the Thirty-Third 
Annual Foriim of the Association for Institutional 
Research held at the Chicago Marriott Downtown, 
Chicago, Illinois, May 16-19, 1993. This paper 
was reviewed by the AIR Forum Publications 
Committee and was judged to be of high quality 
and of interest to others concerned with the 
research of higher education. It has therefore 
been selected to be included in the ERIC Collection 
of Forum Papers. 



Jean Endo 

Chair and Editor 

Forum Publications 

Editorial Advisory Committee 



3 

ERJC 



Assessment of study-abroad programs 

2 

ASSESSMENT OF STUDY-ABROAD PROGRAMS USING 
SURVEYS OF STUDENT PARTICIPANTS 

ABSTRACT 

Study-abroad programs are among the many academic programs being 
evaluated in the assessment of colleges and universities. One evaluative tool is the 
survey of students who participate in these programs. This paper discusses the 
use of these surveys during the past decade to gather information regarding 
program strengths and weaknesses and assess thB impact of the program on the 
students who participated. Institutional researchers at colleges and universities 
where a significant number of students participate in study abroad and/or whose 
campuses sponsor study-abroad programs will find this paper useful. 
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ASSESSMENT OF STUDY-ABROAD PROGRAMS USING 
SURVEYS OF STUDENT PARTICIPANTS 

Introd uction 

It is estimated that one in ten undergraduates in American colleges and 
universities study abroad during their college experience (Cohen, 1987, p. 41), 
There are 1,300 programs sponsored by accredited colleges and universities 
(Howard, 1990, p. v), running the gamut from branch/extension campuses 
operated by North American colleges and universities to co-exchange programs 
with existing institutions in the host countries. Since their experiences often are 
cited by those who studied abroad as being among the most formative in their 
college experience, it seems appropriate in this era of program and outcomes 
assessment to include the evaluation of these programs in the assessment program 
of colleges and universities. Further, if participation in these programs has an 
effect on the interests and values of the students who spend a term or two abroad, 
these should be identified and noted as program outcomes. 

The literature of program review and assessment (eg., Ewell, 1988; Kinnick, 
1985; Wilson, 1987) emphasizes the importance of evaluation of institutional 
effectiveness at the program level. While program review usually is considered to 
be an assessment of a curricular program or department, it can also be applied to 
such services as study-abroad programs. 

Stutzman (1985) has provided a guide to on-site evaluation of undergraduate 
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programs, noting that these programs typically receive little attention from 
evaluators. Among the data collected in these site visits are evaluations from 
student participants, both present and past. 

The use of student surveys for gathering assessment outcomes information 
is a widely-used method for some of the information needed in conducting program 
reviews (Ewell, 1987, p. 20). Indeed, the practice of collecting student evaluation 
data and widely disseminating the results is seen as strengthening the quality of 
education (National Institute of Education, 1984). 

Pascarella and Terenzini (1991) note limited research investigating effects of 
study abroad as inconsistent and "not distinguished for the rigor of its research 
designs and methods" (p. 306). While there is evidence of limited impact, they 
suggest introduction of pre-study-abroad controls might yield very different results. 

A summary of this previous research in Kauffmann, Martin, and Weaver 
(1992) also provides case studies to "attach the research results to reality and 
make them more understandable." (p. 2) These authors also propose a theoretical 
framework for future research on the effects of study abroad on students. 

This paper provides a rationale for conducting the assessment of study- 
abroad programs, gives a report of how one college evaluates the experience of the 
abroad study of its students (both on the overseas branch campus and in co- 
exchange programs), evaluates the immediate effect of study abroad, and makes 
recommendation for future research in this area. In so doing, it is hoped that other 
institutional researchers will benefit from the presenter's experience. 
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Methodology 

The study abroad evaluation process described is part of the assessment of 
the study-abroad programs officially sponsored by Saint Mary's College, a Catholic 
women's college. Saint Mary's is a competitive, highly ranked comprenensive 
college in the liberal arts tradition, sponsoring study abroad programs in Rome 
(overseas branch campus), India (a "Sf mester Around the World" study tour of the 
Far East with several weeks at Stella Maris College), Ireland (students enroll at 
Saint Patrick's College), as well as a summer program in London. 

Periodic surveys of student participants collect evaluative data regarding 
each of these programs (Table 1 identifies the evaluations conducted during the 
past ten years). The evaluation cycle also includes a survey of students whose 
study abroad was in programs not sponsored by the College. 

These surveys are administered to participating students at the end of their 
study abroad, either just before they leave or shortly after chey arrive home. Items 
included on the surveys evaluate the academic program, program advising and 
orientation, personal adjustments, cultural contacts, living arrangements, and 
growth towards personal and institutional goals (sample surveys can be obtained 
from the author). Space is provided on the survey instrument for respondents' 
comments to amplify and/or explain the quantitative responses or to provide 
anecdotal evidence to describe their feelings and experiences. 

Response rates to these surveys generally run about 60 to 80 percent, and 
are generally considered representative of the cohort; indeed, if there is any 

7 
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Table 1 

History of study abroad evaluations at Saint Mary's College 







Ireland 




India 






Rome 






Survey Program 
Respon- Parti- 
dents cipants 


Resp 

rici Lc 


Survey Program 
Respon- Parti- 
dents cioants 


Resp 
Rate 


Survey Program 
Respon- Parti- 
dents cipants 


Resp 
Rate 


1982-83 


18 


25 


72% 












1 984-85 












47 


50 


94% 


1985-86 


18 


28 


64% 












1 987-88 












58 


60 


97% 


1988-89 


14 


30 


47% 












1989-90 


20 


26 


77% 


16 21 


76% 


62 


78 


79% 


1990-91 












35 


59 


59% 


1991-92 


14 


21 


67% 


16 20 


80% 


35 


40 


88% 



response bias, it is the students who have the strongest reactions to their 
experience who are more likely to respond. 

In addition to the student surveys administered to those completing their 
study abroad-program, the College routinely collects information from incoming 
freshmen and outcome measures during and after the college experience. Thus, it 
should be possible to compare pre- and post-experience scores on the same items 
between students whose undergraduate experience included study abroad and the 
others in their cohort who did not study abroad. 
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Results 

The results of the student evaluations show the study abroad experience has 
been a positive one for participants. Particularly, it has opened their eyes to a 
world to which most were not familiar prior to their travel and study abroad. 

While the survey for each program has unique items relevant for the 
particular program, there are common questions on all three surveys that provide 
the basis for comparison between the programs. Among these items are a series 
of questions based on the College's goals for students in these programs. 
Respondents are asked to identify the degree of growth (using a four-point scale 
from ''None" to "Great") they experienced towards each of these goals {see 
Table 2). 

The items on which they reported the greatest amount of growth were 
''Appreciation and understanding of different cultures" (an average of 88.1% of the 
respondents reported experiencing a "great" amount of growth), "Growth in 
independence and maturity" (84.9%), "Greater self-awareness" (80.9%), "Greater 
tolerance for different people and ideas" (75.8%), and "Growth in interpersonal 
skills, ability to interact with others, handle stress, etc." (69.3%). 

An item included only on the India and Ireland evaluations was an area of 
"great growth" by 92 percent of the respondents- "Increased ability to adapt to 
the unfamiliar", and two items included only on the Ireland survey were areas of 
"great growth" by more than 90 percent of the respondents— ''Opportunity to 
travel and see new places" (97.5%) and "Interact with the Irish people" (91.5%). 
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Table 2 

Growth towards College's goals for program participation 





India 




Ireland 




Rome 










88- 


89- 


91- 


89- 


90- 


91- 


Percent reoortina "Great Growth" 


89 


91 


89 


90 


92 


80 


91 


92 


Appreciation/understanding different cultures 


87%100% 


86% 


. 95% 


86% 


87% 


87% 


77% 


Greater tolerance different people/ideas 


62 


100 


83 


67 


92 


66 


67 


69 


Better perspective American culture/history 


50 


87 


46 


50 


46 


48 


46 


26 


Better undfirstanding US/others role in world 


71 


93 


67 


61 


43 


43 


57 


33 


Better understanding other political systems 


67 


71 


50 


13 


42 


34 


29 


23 


Greater se!f-awa?;:ness 


93 


80 


86 


84 


85 


77 


73 


69 


Growth in independence and maturity 


100 


100 


92 


84 


93 


68 


73 


69 


Growth in interpersonal skills 


82 


80 


75 


68 


79 


58 


55 


57 


Inteilectual stimulation/development 


79 


83 


39 


35 


40 


54 


44 


37 


Aesthetic appreciation and enjoyment 


69 


92 


75 


75 


0 


79 


69 


66 


Increased abilitv to use foreion lanoudQe 


NA 


NA 


NA 


NA 


NA 


44 


37 


34 


Appreciation of other cultures/religions 


99 


100 


NA 


NA 


NA 


NA 


NA 


NA 


Deeper appreciation own religious heritage 


38 


80 


40 


36 


13 


39 


27 


20 


Increased ability adapting unfamiliar 


100 


100 


85 


75 


100 


NA 


NA 


NA 


Experience different teaching/education 


NA 


NA 


NA 


33 


50 


NA 


NA 


NA 


Interact with inaigenous (Irish) people 


NA 


NA 


NA 


90 


93 


NA 


NA 


NA 


Opportunity to travel and see new places 


NA 


NA 


NA 


95 


100 


NA 


NA 


NA 


Learn more re family heritage/visit relatives 


NA 


NA 


NA 


53 


40 


NA 


NA 


NA 



(Scale: 1 ="None" 2 = "Little" 3 = "Moderate" 4= "Great") 

Continued 
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Table 2 - Continued 



As a contribution to your total college eCocation experience how would you rate your experience on 
the Program? 

India 

Autumn 1989 4,00 
Autumn 1991 4.00 

Ireland (quostion not used after 1 988-89) 

1982-83 3.89 
1985-86 3.94 
1988-89 3.93 

(Scale: 1 - "Not valuable" 2 = "Of some value" 3 = "Valuable" 4 -"Very valuable") 



The surveys of the India and (some of the) Ireland programs also included an 
overall summative question, "As a contribution to your total college education 
experience, how would you rate your experience on the Program?" (see Table 2). 
On the four-point scale, the respondents were nearly unanimous in describing their 
experience as being "Very valuable". 

Perhaps the most telling results from the survey have been the added 
comments of participants. Following are excerpts from some of these comments: 
The Semester Around the World Program primarily changed my view of 

myself. I learned what I was made of and what I could do. I think beyond exploring 

the Far East, I really explored myself and my independence. (India 1989) 

All of these areas of growth and maturity were possible through the program 
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experience. It was not so much the "biQ" events that were influential, but the day- 
in/day-out routines and relationships that gave the opportunity to grow and 
appreciate. (Ireland 1991-1992) 

I've definitely grov^n spiritually, emotionally, and have a greater love for the 
arts. (Rome 1989-1990) 

I've never learned more about myself and my goals as an adult than all I 
have learned this year, (Rome 1990-1991) 

Living in Rome has had a profound effect on my life— it has given me a 
deeper appreciation for other people and for life in general. (Rome 1990-199 1) 

I don't believe there is any experience that has touched my life in so many 
waysll My Semester Around the World experience has changed my attitudes, ideas, 
outlooks, goals, and life/I Each and every day I feel the positive impact of this 
wonderful experience, (India 1989) 

The entire experience helped me to grow, expand my horizons, deal with 
people better, realize my independence and gain an overall confidence in myself and 
my abilities to adapt, learn and grow in a new environment (India 1989) 

I grew more (socially, culturally, independently) during my year of study in 
Ireland than I ever have before. (Ireland 1988-1989) 

I strongly believe that study abroad is a vital part of a liberal arts education, 
especially in what one learns outside of the classroom. (Ireland 1988-1989) 

The opportunity to live in and study in another culture is one of the most 
rewarding experiences a person could ever have. (Ireland 1939- 1990) 

This program opened my mind to the rest of the world. It gave me a better 
understanding of the people in the world. It also taught be aspects about life and 
death, poverty and wealth, and what is accepted and not accepted throughout the 



ERLC 



12 



Assessment of study-abroad programs 

11 

world. This program helped me to become a well-rounded person, (India 1991) 

The program taught me more than any other semester I've encountered. . . 
[TJhere are still tons of things I have yet to learn and understand. (India 1991) 
Saint Mary's conducts a survey of seniors every other year along with a 
parallel sophomore survey. The senior survey includes a items asking respondents 
to evaluate the degree to which participation in a study abroad program was of 
value to their overall intellectual and personal growth (see Table 3). Although the 
scale was changed in 1991 to a truer Likert-type scale, the respondents are nearly 
unanimous in their perception ol the value of their study abroad program. 

The last two senior surveys also included an open-ended question asking 
respondents to describe their most successful experience while at Saint Mary's. 
While the 1993 survey is still being processed at the writing of this paper, the 
study abroad experience was cited by more of the 1991 respondents than any 
other experience except for their overall academic achievement (other topics 
mentioned include personal development and independence, extra-curricular 
activities and/or athletics, student-faculty interaction, and relationships with other 
students). Excerpts from some of these comments follow: 

/ learned more about myself, about life, about people in general than I ever 
could have at home. (Ireland participant) 

Through [the Rome Program] I was able to increase my knowledge and 
awareness about myself and the person I would like to become. 

Beyond a shadow of a doubt, [the Rome Program] was the most important 
four months of my life. . . —opportunities and experiences have shaped me forever. 
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Table 3 

Value of program to overall intellectual and personal growth - Senior Survey 



Year Score 
1985 3.94 
1987 3.99 
1989 3.94 
(Scale: 1 = "Negative factor" 2 = "None" 3 = "Some" 4 = "Great") 

1991 4.72 
1993 4.75 
(Scale: 1 ="Very little" . . . 5 = "Very much") 



Included on the senior survey are a number of items regarding respondents' 
growth while in college. Table 4 lists a selection of these items, picked for their 
potential relationship with the expected outcomes of study abroad. Also shown 
are the average scores on these items for the past two survey administrations, 
grouped by whether the respondent participated in a study abroad program or not. 
Independent sample t-tests or chi-square analyses were used to determine whether 
or not there were significant differences in these scores. Some items were also 
included on this list because differences were found between those who were 
study abroad participants and the other respondents. 

There was hardly any consistency across the two surveys on the items 
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Table 4 

Results of senior survey items by participation status 



1991 



1993 



Prooram Darticioation? 


No 


Yes 


No 


Yes 


Self-rating, growth over four years: 










Intellectual self-confidence 


4.05 


4.13 


4.33 


4.49 


(Scale: 1 ="Much weaker" . . . 5 = "Much stronger") 










Importance personallY: 










Promote racial understanding 


2.72 


2.87 


2.86 


3.14 


Participate in a community service program 


2.64 


2.80 


2.81 


3.11" 


Influence social values 


2.57 


2./9* 


2.95 


3.00 



(Scale: 1 =''Not important" 2 = "Somewhat important" 3 = "Very important" 4 = "Essential") 
Extent to which experience growth over four years: 



Ability to think clearly and critically 


4.09 


4.17 


4.15 


4.40** 


Understanding and tolerance for other people and ideas 


4.08 


4.11 


4.02 


4.34** 


Independence, ability to make own decisions 


4.35 


4.24 


4.29 


4.42 


Formulating the values and goals of my life 


3.92 


4.14* 


4.19 


4.38 


Interest in and desire for life-long learning 


4.03 


4.25 


4.04 


4.37* 


Social skills, ability to get along with others 


3.88 


3.81 


4.09 


4.24 


Development of self -identity and self-confidence 


4.22 


4.22 


4.24 


4.51* 


(Scale: 1 =''None'' . . . 5 = "Very much") 










Achievement of Saint Mary's objectives: 










Keen self-knowledge 


3.78 


3.99' 


4.01 


4.21 


Intellectual interests 


4.17 


4.22 


4.19 


4.42* 


Cultural interests 


3.74 


A.06* 


3.81 


3.99 


Competence in quantitative skills 


3.88 


3.55** 


3.92 


3.87 


Competence in modern languages 


2.79 


3.35*** 


2.80 


3.29** 


(Scale: 1 ="Verv little" . . . 5 = "Very much") 










Number 


152 


85 


147 


70 



•**p < ,001 **p < .01 *p < .05 



Continued 
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Table 4 - Continued 



« « 



1991 1993 

Program participation ? No Yes No Yes 

Extent to which Saint Mary's had effect: 

Knowledge of contribution of women to history/culture 3.1 1 3.22 3.36 3.83** 

Awareness of women's rights/responsibilities today 3.74 3.74 3.92 4.30 

Aware of need for responsibility in global environment 3.66 3.75 3.68 3.81 

Knowledge of problems/needs of the third world 2.95 3.18 3.05 3.39* 
(Scale: 1 =''No change" . . . 5 = "Very much") 

Satisfaction with: 

Tone and quality of intellectuallife in classroom 4.14 4.03 4.18 4.40* 
(Scale: 1 =''Very dissatisfied" . . . 5 = "Very satisfied") 

Agreement with: 

More interested in ideas and intellectual matter 4.16 4.25 4.13 4.37* 

Developed close relationships with other students 4.66 4.67 4.56 4.80* 

Few faculty spend time out of class with students 2.00 1 .86 1 .89 1 .60* 

Few faculty are outstanding or superior 1 .90 2.11 2.04 1.74* 

Getting good grades is not important to me 1 .83 2.00 1.87 2.19* 
(Scale: 1 = "Strongly disagree" . . . 5 = "Strongly agree") 

Importance of 

Understanding and tolerance for other people and ideas 50% 49% 48% 59% 

Independence, ability to make own decisions 63% 62% 52% 63% 

Social skills, ability to get along with others 44% 40% 39% 46% 

General tone/quality of intellectual life in the classroom 50% 38% 30% 46%* 

Arrangements for expressing student opinion on policies 34% 28% 24% 40%* 
(Scale: 1 = "Personally important" 0 = "Not important") 

Number 152 85 147 70 

***p < .001 **p < .01 •p < .05 
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which were significantly different between those who participated in study abroad 
programs and those who did not. The only item which was significant (p < .05) 
both years was "Competence in modern languages". 

Even more surprising were items identifying areas where the study abroad 
experience might have been expected to produce differences and there were no 
significant (p < .05) differences. These include growth in "Intellectual self- 
confidence", "Independence, ability to make own decisions", and "Social skills, 
ability to get along with others"; the extent to which the College had an effect on 
''Awareness of need for responsibility in global environment"; the importance of 
"Understanding and tolerance for other people and ideas" and "Independence, 
ability to make own decisions". 

While in general the observed scores of respondents who had participated in 
study abroad and those who had not were more positive for f ase who had studied 
abroad, there were two items across both years where this was not the case. The 
degree to which respondents felt they had achieved the College's objective for 
"Competence in quantitative skills" was lower for those who studied abroad than it 
was for those who had not. The respondents who had studied abroad were less 
likely to disagree with the statement "Getting good grades is not important to me". 

A comparison of the scores for the seniors in 1991 with that of the seniors 
in 1993 shows the 1991 respondents tended to be more negative in their rating of 
their growth and experiences at Saint Mary's than we/e the 1993 respondents. 
Since the instruments used in the senior and sophomore surveys include 
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many items in common, it is possible for analyses of these surveys to include the 
study of change over time (sophomore to senior years) in respondents' opinions, 
attitudes, and perceptions of their gro>Arth as students. Indeed, one of the initial 
plans for this study was to make comparisons using many of the items found on 
Table 4 between those who had study abroad experience and those who did not. 
However, it was discovered this analysifi is impossible to do at Saint Mary's, since 
most of the study abroad experience occurs during the sophomore year and those 
who are abroad are not included in the sophomore survey population. 

However, it was possible on a few items on the senior survey to compare 
responses of seniors with their responses on the ACE/CIRP freshman survey they 
had taken as entering freshmen. These results are shown on Table 5. The 
statistical procedure used in this analysis was Analysis of Variance (ANOVA), using 
the equivalent freshman survey item as a covariate. Hence, the test of significance 
measures whether or not there is a difference between the scores of the seniors on 
each item, controlling for their scores on the equivalent freshman survey item. 

The only items for which this procedure was possible were four items 
evaluating the seniors' perception of growth over their four years of college 
experience (as freshmen they had identified their ranking on these items compared 
with their peers) and twelve items on which they indicated (both as freshmen and 
as seniors) the degree to which each item had personal importance. 

There were no items with significant differences (p < .05) for both years 
between the scores of those who had participated in study abroad and those who 

13 
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Table 5 

Differences between senior survey items by participation status, 
controlling for freshmen survey scores 



1991 1993 

Program participation ? No Yes No Yes 

Self-rating, growth over four years: 

Drive to achieve 3.94 4,11 4.15 4.00 

Intellectual self-confidence 4.38 4.48 4.20 4.57' 

Social self-confidence 4.1 6 4.09 4.07 4,24 

Leadership ability 4.07 3.98 4.07 4.22 
(Scale: 1 =''Much weaker" . . . 5 = "Much stronger") 



Importance personally: 



Help others in difficulty 


3.31 


3.13 


3.39 


3.53 


Raise a family 


3.24 


3.40 


3.44 


3.33 


Develop a meaningful philosophy of life 


2.90 


3.15 


3.15 


3.22 


Obtain recognition fr colleagues for contributions to field 


2.59 


2.45 


2.57 


2.69 


Promote racial understanding 


2.72 


2.81 


2.78 


3.W 


Participate in a community action program 


2.66 


2.80 


2.81 


3.06 


Be authority in my field 


2.52 


2.55 


2.56 


2.61 


Be involved in programs to clean up the environment 


2.66 


2.73 


2.81 


2.67 


Be very well off financially 


2.25 


2.25 


2.33 


2.22 


Have administrative responsibility for the work of others 


2.18 


2.13 


2.29 


2.31 


Achieve in a performing or creative art# 


1.69 


2.00 


1.59 


1.97 


Influence social values 


2.48 


2.84** 


2.83 


3.03 


(Scale: 1 ="Not important" 2 = "Somewhat important 


" 3 = "Very important" 4 = "Essential") 


Number 


89 


55 


61 


37 



#Senior survey item read "Achieve in a performing or creative art", covariates were freshman survey 
items "Becoming accomplished in one of the performing arts (acting, dancing, etc." and "Creating 
artistic work (painting, sculpture, decorating, etc.)". 

••p < .01 *p < .05 

.^9 
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had not, and only three items which were significant for one of the two classes. 
Seniors in 1991 who had participated in study abroad were more likely to value 
personally the importance of "Influencing social values" than were the seniors who 
had not studied abroad, while seniors in 1993 who had participated in study abroad 
were more likely to have experienced growth in their "Intellectual self-confidence" 
and to rate as more important their plans to ''Promote racial understanding" than 
were the seniors who had not studied abroad. 

While many of these items would not be expected to be related to the 
effects of study abroad, the item regarding the personal importance to ''Promote 
racial understanding" could be considered to have such a relationship, with growth 
in this area being one of the desired outcomes of the study abroad programs. 
However, no significant differences were found on this item. 

Discussion 

The quantitative and verbal responses from each of the program evaluations 
provide compelling evidence of the immediate effect of the program on the 
participants, and this is to be expected. When the surveys are being completed, 
the respondents are either still abroad, or have just returned and are encountering 
the "return shock" experienced by those who have been away from campus and 
country for a period of time. The stimuli of their experience are fresh, and yes, 
indeed they do feel that being abroad has had a major impact on them. 

But evidence of the long-term effect of this study abroad is more difficult to 
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find. Certainly it is expected that participants in a study abroad program where the 
language of instruction and daily living is not English would become more 
competent in that language, and it is possible that the effect of Saint Mary's 
program in Rome (the only one of the College's three programs where there is 
serious opportunity to use a different language) is strong enough to affect the 
overall results for all the study abroad participants. The finding that study abroad 
participants were less likely to achieve competency in quantitative skills probably 
reflects the greater likelihood of program participants to come from liberal arts and 
humanities and less likely to come from the sciences and/or mathematics, due not 
only to the interests of students in these disparate disciplines, but also possibly to 
the lack of flexibility in certain curricula. 

Reasons as to why no differences were found where they might have been 
expected probably can be associated with the fact that these are items not 
specifically addressing the expected outcomes of study abroad, but rather are only 
tangential to whatever effect there is. 

In terms of their value in assessing the programs, note that the chief 
academic officer at Saint Mary's identifies changes in the manner that the College 
administers its study-abroad programs, based at least in part on the results of these 
evaluations. The types of materials and topics used during the on-campus pre- 
experience orientations have been modified, and some of the pricing and 
programming related to the Rome program have also been adjusted based on 
student comments; the feedback received is valued. 
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Conclusions 

After reviewing the results of several years of conducting surveys of study 
abroad participants, it is apparent that these studies fulfill an important part of 
evaluating the College's study-abroad programs. The information obtained is not 
only useful in determining each program's strengths and weaknesses, but it also 
identifies areas of personal growth in a number of areas both academic and social. 

The difficulty in ascertaining whether or not study abroad programs had 
long-term effects on their participants can be attributed in part to methodological 
problems related to this specific study. Since the routine sophomore survey is an 
inappropriate vehicle for collection pre-experience information, other means of 
gathering such data (including data from program non-participants) need to be 
explored. At the present time, it appears at Saint Mary's that the best mechanism 
for gathering pre-experience information may be the freshman survey, administered 
during their initial orientation to the campus. Further study is planned in this area. 

Nonetheless, the results presented here should contribute in a small way to 
the growing body of research on the effect of study abroad on student interest and 
values, and institutional researchers from campuses sponsoring and/or encouraging 
study abroad can benefit from the information presented and recommendations 
suggested in this paper. 
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